Provably Efficient Reinforcement Learning in Decentralized General-Sum Markov Games

This paper addresses the problem of learning an equilibrium efficiently in general-sum Markov games through decentralized multi-agent reinforcement learning. Given fundamental difficulty calculating a Nash (NE), we instead aim at finding coarse correlated (CCE), solution concept that generalizes NE by allowing possible correlations among agents’ strategies. We propose algorithm which each agent independently runs optimistic V-learning (a variant Q-learning) to explore unknown environment, while using stabilized online mirror descent (OMD) subroutine for policy updates. show agents can find $$\epsilon $$ -approximate CCE most $$\widetilde{O}( H^6S A /\epsilon ^2)$$ episodes, where S is number states, size largest individual action space, and H length episode. appears be first sample complexity result generic games. Our results rely on novel investigation anytime high-probability regret bound OMD with dynamic rate weighted regret, would independent interest. One key feature our it decentralized, sense has access only its local information, completely oblivious presence others. way, readily scale up arbitrary agents, without suffering from exponential dependence agents.